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framework to help educators and/or trainers select quality 
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Abstract 



Evaluating and selecting the appropriate software is a very important component of success for using 
multimedia systems in both educational and corporate settings. This paper attempts to explore the issues surrounding 
the evaluation of multimedia software and to present a comprehensive criteria for evaluating and selecting multimedia 
software for effective instruction, from general software evaluation criteria to the specialized multimedia evaluation 
criteria. An integrated checklist for multimedia software evaluation was included. {Key Words: Computer-Mediated 
Multimedia (CMM), Software, Criteria Selection, Evaluation, Instruction) 

INTRODUCTION 

The use of microcomputers for education and training in schools and homes as well as in corporate settings are 
commonplace since relatively inexpensive microcomputers came on the market as an instructional tool. Many 
educational professionals believe that in the world of computers, media, and digital technologies, multimedia would play 
a large part in educational reform (Galbreath, 1992; Jost & Schneberger, 1994), and the software available for computer- 

mediated multimedia (CMM) 13 has been increasing both in quantity and in quality over the past couple of years. 

Few instructional software programs, however, are field tested with actual students prior to distribution (Dudley- 
Marling & Owston, 1987; Heller, 1991). The ways in which these software are used vary with the context of their use, 
with different age levels, subject areas or classroom setting. Observers of instructional software have cited a number of 
specific problems with courseware, including technical inadequacy, poor pedagogy, amateurish programming, and 
inadequate documentation (Heller, 1991). 

CMM system is an entirely new kind of media experience bom from TV and computer technologies. It can be 
defined as the integration of two or more communication media, controlled or manipulated by the user via a computer, to 
present information (Cotton & Oliver, 1993; Holsinger, 1994; Galbreath, 1992; Poncelet & Proctor, 1993; Tolhurst, 
1995). CMM program is a powerful combination of text, images, animation, sound, color, and video in a single, 
computer-controlled presentation. CMM entails "the use of the computer to integrate and control electronic media devices 
such as monitors, videodisc players, CD-ROM players, and other electronic equipment” (Poncelet & Proctor, 1993, p. 

93), and offers random access. Evaluation and Selection of CMM systems, therefore, may be different from computer- 
mediated instruction (CMI). 

Instructors, as well as parents, often have difficulty distinguishing quality instructional software from the trivial 
and ineffective software now on the market. Guidelines and sources of knowledge about evaluating and selecting quality 
software program are limited (Chang & Osguthorpe, 1987). As more of the instructional software is placed on the 
market, the need for careful review of the material prior to its purchase becomes increasingly necessary. CMM software, 
like all other educational material, should be evaluated with a thorough and detailed evaluation before it is used in the 
classroom (Heller, 1991). 

According to the report of Educational Products Information Exchange (EPIE) (Taylor, 1987), only about a 
quarter of available software for elementary and secondary schools have been evaluated adequately. Neill and Neill (1993) 
also report that only 7.7% of the total reviewed software as 'worth of an A grade' using their criteria for quality assurance. 
This means that the end-users of computer softwares would have to conduct its own evaluation of three-quarters of the 
instructional software program it is considering. There appears to be a continuing need, therefore, for an approach to 
evaluating software which is general enough to be widely applicable but specific enough to provide the kind for 
information that will allow decisions to be made about the acceptability of software programs under reasonable 
consideration. 

The purpose of this paper is to explore the issues surrounding the evaluation of multimedia software and to 
present a comprehensive criteria for evaluating and selecting CMM software for effective instruction, from general 
software evaluation criteria to the specialized multimedia evaluation criteria. 



1 3 Computer mediated multimedia systems incorpoate "the computer as a display device, management tool, and/or source of 
text, pictures, graphics, and sound" ( Heinich, Molenda, Russell, & Smaldino, 1996, p. 260). 



O 



414 



J 



EVALUATION AND SELECTION CRITERIA 



Evaluation assumes a value judgment. When evaluating and selecting an instructional software, there are some 
general principles that apply to all categories of software. Almost every product-oriented evaluation includes some items 
about the use of the program. These aspects considered for evaluating instructional software might include such things as 
contents, curriculum issues, design factors, technology factors, human factors, documentation and packaging, 
availability of support materials, classroom management, vendor factors, cost factors, and so on (Gray, 1991; Gros & 
Spector, 1994; Heller, 1991; Knight, 1992; Reiser & Dick, 1990; Tolhurst, 1992). 

Even though most of these checklist items were created for evaluating traditional CMI, similar proposals have 
been made for the evaluation of CMM software (Atkins, 1993; Gros Sc Spector, 1994; Hutchings et al., 1992; Knight, 
1992). CMM software is quite different from traditional CMI in terms of linking display tools, management tool, and/or 
design elements such as text, graphical images, animation and sound. For this reason, new items of evaluation should be 
added. Nielsen (1990) proposed five additional parameters for evaluating the usability of multimedia software: ease of 
initial learning, efficiency of use, ease of remembering interface items, error rates, and subject response to the system. 

Well designed CMM software is instructional^ sound materials that take advantage of the unique capabilities of 
the computer. Unfortunately, however, not all programs marketed as CMM are well-designed, instructional^ sound, and 
technically reliable (Sloane, Gordon, Gunn, & Mickelsen, 1989). To make a reasonable decision, criteria should be 
developed to give the evaluator a framework for the task. The most important criterion for CMM software evaluation is 
the consideration which reflects the needs and characteristics of individual teachers and students in using the CMM 
software. 

Teachers can use a formal evaluation of CMM software to guide their initial screening of software, but their 
final judgment should depend upon their own observations of their students as the students interact with the software 
(Dudley-Marling & Owston, 1987). There is no one best CMM software-different software works best for different 
learners and for different instructional purposes, and no one CMM product is ideal for any location that serves diverse 
learners (Sorge, Campell, & Russell, 199 k Criteria considered for evaluating CMM software in the study include 
content issues, instructional design, user interface, and documentation. Evaluation criteria include address both general 
CMI evaluation criteria and the specialized CMM evaluation criteria. 

Content Criteria 

The content presented in CMM software should be appropriate to the its objectives and reflect the level of 
difficulty, sequencing and quantity of content sufficient to meet those objectives. Because objectives are used to plan 
instruction, to facilitate the evaluation of course outcomes, and to prepare and review test items (Taylor, 1987). 
Objectives may affect certain learner characteristics. The target audience, therefore, should be specified, grade level and 
particular characteristics such as gifted, talented, or handicapped (Gold, 1984). The learning objectives should be 
delineated, including how the various features of the instructional design fulfill the learning objectives, and should be 
stated in measurable terms so that the evaluation of lesson will reveal whether the mastery has been achieved. 

The contents and instructions of CMM software should be accurate, fair, and valued. The presentation of CMM 
system content should not only be clear and logical, but be sufficiently simple, complex, technical or non-technical for 
the intended target learners (Bitter & Camuse, 1984; Gold, 1984). 

General Criteria 

• Instructional objectives are clearly stated. 

• Definitions of target audience and prerequisite skills are stated. 

• Contents and vocabulary level for intended users are appropriate. 

• Specific information presented is accurate and answers are correct. 

• Sequence of lesson information and instructions are logical and clear. 

• Contents of cultural, sexual, and other stereotypes are free. 

Specialized Multimedia Criteria 

• Contents of text, image, animation, sound, video, etc. are congruent to CMM information. 

• Terms or words in the CMM software are chosen appropriate for linking multimedia design elements (Tolhurst, 
1992). 
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Instructional Design Criteria 



Interaction activities in educational setting not only maintain learners’ attention and increase their involvement 
on learning task, but also result in better performance on knowledge and/or skills. One of the essential features of CMM 
software in contrast to some other instructional media is its capacity to require and act upon learners' interaction. 
Researchers emphasize this important aspect of CMI, however, many commercial CMI software fall short in the 
characteristics (Alessi & Trollip, 1991). Designing interactions which are frequent, relevant, and increase learning is 
harder than even experienced CMM developers believe. Software evaluator should consider learner involvement as an 
important for evaluating CMM software. 

Anothei advantage of CMM is its capability to individualize students’ learning. The ability of instructional 
programs to adapt to individual needs rests upon the type of individual diagnosis they are capable of doing. Adaptive 
instruction must be capable of: (a) gathering diagnosis information; (b) inferring from this information the specific needs 
of the students, and (c) adjusting instruction accordingly (Venezky & Osin, 1991). Good CMM software will adapt to 
the learner, capitalizing upon his/her talents, giving extra help where the learners is weak, and providing motivators each 
learner responds to. However, most commercial software works about the same for all learners. Matching learners up 
with appropriate lessons and methodologies is important design factor of CMM software. 

Motivation is an essential factor of instruction and learning. Instructional designers, therefore, should use 
appropriate motivational strategies for age level, and social and/or cultural background of die learner. Several motivation 
researches on CMI support that CMM techniques enhance learners’ motivation and result in better learning. For the 
design of CMM programs, designers should first judge the degree of intrinsic motivation a course might tap in the target 
audience, and then decide how much, if any, extrinsic motivation is needed. The options for extrinsic motivators usually 
include game formats, humor, use of personal name for variation in drill formats and feedback on progress (Venezky & 
Osin, 1991). 

Feedback is the reaction of a CMM software to the learners' response and may take many forms, including text 
message and graphic illustration. Its primary function is to inform the learner of the appropriateness of a response. There 
are two kinds of feedbacks: informative and motivational feedback. Informative feedback enables the learner to take 
corrective actions regarding behavior deficits, while motivational feedback is to increase the likelihood that the learner 
will continue to emit the behavior. Thus, motivational feedback is a form of reinforcement and most CMI researches on 
feedback support the effectiveness of feedback on learning (Taylor, 1987). To be a good CMM software, it can generate, 
store, and utilize appropriate feedback about learner needs. 

General Criteria 

• Learners are actively involved with interaction. 

• Availability of varying levels of difficulty is provided according to the skill level of the learner. 

• Motivational factors are included in CMM software. 

• Appropriate feedback is provided. 

Specialized Multimedia Criteria 

• Learner can stop and move to different places in the CMM program without repetition. 

• CMM software encourages a guided discovery or discovery mode of learning (Tolhurst, 1992). 

User Interface Criteria 

One of the claims about CMM software is that the user is free to explore paths through the combinational 
information such as text, graphics, sound, animation, video, etc. under computer control. The main concern of user 
interface criteria is always for an easy-to-use, easy-to-install, and easy-to-transport system (Knight, 1992). The evaluation 
of a piece of CMM software components can be guided by these inquiries topics: ease of use, clarity of directions and 
responses, simple error handling, screen design, learner control, and record keeping (Chang & Osguthorpe, 1987; DeJoy 
& Mills, 1989; Heller, 1991; Knight, 1992; Tolhurst, 1992). 

CMM software installation should not require the service expert, but be able to be done on a hard disk. The 
steps should be clear and well defined. Tutorials should be included to aid the quick mastery of how the software works. 
One should be able to learn the basics in a short time. The CMM software needs to be branched in such a way as to 
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allow sophisticated users to skip over the basic directions. After mastering the mechanics of using the CMM software, a 
learner will want to be able to streamline the commands. So frequent users can use shortcut such as abbreviations, special 
keys, hidden commands, macro facilities, and so on. Consistent sequences of actions should be required in similar 
situations and consistent commands should be employed throughout (Shneiderman, 1992). 

Directions on how to operate the CMM software should be given in language appropriate to the target audience. 
Language, sentence length and complexity, aiid style of communication to the user should be clear and appropriate to the 
capabilities of the intended audience. The required reading level of the intended user must be respected (Gold, 1984). 

Basically, there is lack of systems error in CMM software, however, learners occasionally hit the wrong key(s) 
by mistake. Even designing error-free system is best, it is very difficult job. If an error is made, it is hoped that nothing 
harmful, such as losing one's data, will occur. System should detect the error and offer simple, comprehensive 
mechanisms for handling the error (Shneiderman, 1992). The user should be signaled when a wrong key is hit. The 
learner can repair only the faulty part instead of retyping the entire command. Erroneous commands should leave the 
CMM system state unchanged, or the system should give instruction about restoring the state. 

It is essential that the instructional information be formatted on the CMM screen for easy reading. Its display 
should keep clean, simple, attractive, and aesthetic balance using screen design elements such as text, image, animation, 
sound, color, video, etc. The use of visual cues can be effective in gaining and keeping the learners' attention during 
instruction (Faiola & DeBloois, 1988; Poncelet & Proctor, 1993). The use of screen design elements should be 
appropriate to CMM software and enhance the learning process and results. 

Learner control is a crucial design variable in all CMI software. This means whether control of sequence, 
content, methodology, and other instructional factors is determined by the learners, the lesson, or some combination of 
two (Alessi & Trollip, 1991). In reality, all lessons have a mixture of learners and lesson control. The whole notion of 
interactivity is realized when course allows the learners to weave his/her own educational environment. This can be done 
by granting the learner control over certain aspects of the CMM software. 

If CMM software is to utilize the full capabilities of computer effectively, it should have a sophisticated 
management systems. One of the most unique and useful capabilities of CMM software is its ability to keep track 
automatically of learner progress through software materials. If CMM software is equipped with this capacity, it will be 
much benefit for teachers (Truett & Gillespie, 1984). Good record keeping abilities should allow for the tracking of an 
entire class of learners, not merely the accounting of one learner's work. 

General C riteria 

• Management system of CMM software is easy to use and flexible. 

• Screen displays are easy to read (text size/color/spacing). 

• Screen design elements such as text, image, sound; animation and color combine to enhance, not distract from 
information presentation. 

• CMM software allows the learner to correct his/her error. Error messages are helpful and user-friendly. 

• Screen display should be kept clean, simple, attractive as well as aesthetically balanced. Special effects are used 

effectively and not overdone. 

• Learners have control over the rate of presentation and/or navigation. 

• A clear and useful summary of learners' activities and progress is provided. 

Specialized Multimedia Criteria 

• Multimedia element links and system links are distinguishable (Tolhurst, 1992). 

• CMM software contains different contexts of use (e. g. descriptive text and a glossary). 

• CMM software contains aids to assist learner navigation. 

Documentation Criteria 

Documentation is to help the user install and maintain the CMM software. The purpose of user's manual is to 
inform the learners of two things: the operation of the CMM software and the instruction on software design (Gold, 
1984). User's manual should be included the following: (a) specifications of the hardware configuration, operating 
system and programming language code; (b) definition of any external software required; (c) installation instructions; (d) 
instructions on how to operate the software; and (e) explanation of how to exercise features. 
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The instruction on CMM software design should be written objectives, content, curriculum issues, and 
evaluation. The target audience should be specified as to age, grade level, any particular characteristics such as gifted or 
handicapped. For the knowledge issues, the knowledge and/or skill prerequisite for the CMM software must be clearly 
stated. For evaluation, the CMM software should provide the teacher and the student a way to evaluate learner 
performance and effectiveness (Gold, 1984). 

It is imperative that CMM software be accompanied by a technical manual containing general information, 
functional description, and required features (Gold, 1984). The technical manual should include general information that 
describes the technical features, hardware configuration, operating system and programming language or code. It should 
also contain complete installation and start-up instructions. The manual should contain a functional description including 
an explanation of how each feature works, when it operates, under what conditions and any restrictions that apply to its 
use. It should explain in detail how to exercise each feature. 

General Criteria 

• User’s manual contain detailed and complete indices of the information available in the CMM software.' 

• Commands for operation are consistent and thorough, including the use of multiple diskette or discs (DeJoy & 

Mills, 1989). 

• Directions for installation and operation are clear, accurate and complete (Sorge, Campell, & Russell, 1993). 

• Specifications of hardware requirements, operating system and programming language code are provided. 

• Any requirements for staff support are made clear. 

• An information ’hot line’ is available. 

Specialized Multimedia Criteria 

• The linking maps or descriptions of the hypertext and/or other screen links are provided (Tolhurst, 1992). 

• Any identification or description of the branching techniques used in CMM software is presented. 

CONCLUDING REMARKS 

The focus of this presentation is on discussing the issues surrounding the evaluation of CMM software and on 
building a comprehensive criteria for evaluating and selecting CMM software for effective instruction. These criteria will 
provide a useful framework to help educators and/or trainers select a quality instructional software for their instructional 
purposes. However, there is no widely agreed-upon standards or criteria for CMM software evaluation. It is a paramount 
need to develop not just minimal standards but standards that will act as guidelines to help developers, evaluators, and 
consumers determine what quality CMM softwares are. 
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Appendix 

Multimedia Evaluation Checklist 

General Information Date: 



Software Title: 


Source/Distributor: 




Phone: Fax: 


Subject Area: 


Cost: 


Target Audience: 


Length: 


Minimum Hardware Requirements 


Computer Platform: ( ) Macintosh (H. he, Ilgs) 


( ) IBM ( ) Other 


Memory: ( ) 1M ( ) 2M ( ) 3M ( ) 4M ( ) 5M ( ) 10 M ( ) Other ( ). 






Content 



Criteria 


Rating* j 


Weight** 


General 

• Instructional objectives are clearly stated. 

• Definitions of target audience and prerequisite skills are statu!. 

• Contents and vocabulary level for intended users are appropriate. 

• Specific information presented is accurate and answers are correct. 

• Sequence of lesson information and instructions are logical and clear. 

• Contents of cultural, sexual, and other stereotypes are free. 

Multimedia 

• Contents of text, image, animation, sound, video, etc. are congruent to 
CMM information. 

£ Terms or words in the software are chosen appropriately for linking 
multimedia design elements. 
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Instructional Design Criteria 



Criteria 


Rating* 


Weight** 


General 

• Learners are actively involved with learner interaction. 

• Availability of varying levels of difficulty is provided according to the 
skill level of the learner. 

• Motivational factors are included in software. 

• Appropriate feedback is provided. 

Multimedia 

X Learner can stop and move to different places in the program without 
repetition. 

X Software encourages a guided discovery or discovery mode of learning. 







User Interface Criteria 



Criteria 


Rating* 


Weight** 


General 

• Management system of the software is easy to use and flexible. 

• Screen displays are easy to read (text size/color/spacing). 

• Screen design elements such as text, image, sound, animation and 
color combine to enhance, not distract from information presentation. 

• Software allows the learner to correct his/her error. Error messages are 
helpful and user-friendly. 

• Screen display should be kept clean, simple, attractive as well as 
aesthetically balanced. Special effects are used effectively and not 
overdone. 

• Learners have control over the rate of presentation and/or navigation. 
X A clear and useful summary of learners’ activities and progress is 

provided. 

Multimedia 

• Multimedia element links and system links are distinguishable. 

X Software contains different contexts of use (e.g. descriptive text and a 
glossary). 

X Software contains aids to assist learner navigation. 
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Document Criteria 



Gspgral 



Criteria 



Rating* 



Weight* 



• User s manual contain detailed and complete indices of the information 
available in the software. 

• Commands for operation are consistent and thorough, including the use 
of multiple diskette or discs. 

• Directions for installation and operation are clear, accurate and 
complete. 

• Specifications of hardware requirements, operating system and 
programming language code are provided. 

• Any requirements for staff support are made clear. 

• An information 'hotline' is available. 



Multimedia 

• The maps or descriptions of the hypertext and/or other screen links are 
provided. 

X Any identification or description of the branching techniques used in the 
software is presented. 

* Rating: 3 - High 2 - Medium 1 - Low 

** Weight: 3 - Very Important 2 - Important 1 - Little Important 



Overall Evaluation and Recommendations 





Evaluator Record 




Name: 


Signature: 


Affiliation: 
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